Skip to content

generate the py ISC parser from the parsanol .parg grammar (WO17 step 4b) - #843

Merged
ronaldtse merged 1 commit into
mainfrom
wo17/generated-parser
Nov 11, 2026
Merged

ronaldtse merged 1 commit into
mainfrom
wo17/generated-parser

Conversation

@ronaldtse

Copy link
Copy Markdown
Contributor

Summary

WO17 step 4b, python leg: replace the hand-written recursive-descent ISC parser (interscript-py's isc.py _Parser) with a parser generated from the canonical parsanol grammar, mirroring the merged TypeScript milestone (interscript-ts PR #128).

  • grammar_source.py — the ruby lib/interscript/isc/grammar/isc.parg text vendored verbatim, sha256-pinned (b0f7a7caf462a1671a80ef974a59425c636f1d32cb3e405280991b21a6edd712, the same pin the ts vendored copy carries); synced by scripts/sync_isc_grammar.py; pin test tests/test_grammar_source.py.
  • parg_reader.py — PARG lexer/parser (Parsanol::PARG subset), ported module-for-module from the ts parg-reader.ts.
  • peg.py — table-driven PEG interpreter + parsanol CanFlatten semantics, plus two py-side additions: from/to match-span recording (the W2 override key is the byte-exact source text of the from atom, which the capture tree alone cannot supply) and failure diagnostics (the two hand-parser error messages the suite pins).
  • actions.py — capture tree to IscDocument, mirroring the replaced hand parser's semantics with per-site DEVIATION comments; drives the unchanged W1/W2 expansion, which stays in isc.py (_Expander, verbatim from the old _Parser) together with the document model and the compiled-map converter. parse_isc's public API is unchanged.
  • Four GRAMMAR_EXTENSIONS, one named block: the three the ts generated parser ships (any_arg item fallback, block-rule outside-brace when-guard, hyphenated run dependency names) plus rule_items accepting use inside parallel/sequence blocks (ISC-WF-1 §3 splice-into-pass; the py suite pins it, the corpus never uses it).

Evidence (all runs on this branch)

  • py-vs-py parity (interscript-py/scripts/parse_parity_py.py, maps at origin/main, 434 files): the driver parses every corpus file with BOTH parsers — 434/434 identical under the maps canonical schema and 434/434 field-for-field raw equality, including byte-exact from_source override keys and the written constraint order.
  • maps harness (scripts/parse-parity.mjs, this branch on the py side): 411/434 — the same score, the same 23 failing files (0 status flips), and the same divergence-kind counts (item-kind 678, constraints 144, item-value 4) as the hand parser on the same-day ts side; the gap is the py normalizer's constraint re-sorting and none-lowering, documented in the harness.
  • py suite: pytest tests/ — 18 failed / 394 passed / 8 skipped; the failure set is identical to a pristine pre-swap run (same 18 test ids, all pre-existing gaps in parser-independent paths: 4 conformance + 14 embedded-map rows). The golden additivity gate still replays byte-identically.
  • lint: ruff clean over the touched files.

Known tradeoff

The table-driven interpreter parses the 434-file corpus in ~86s where the hand parser took ~7s (typical maps still parse in tens of ms; the gap shows only on the megabyte jamo/hanja maps and in the gates that parse the whole corpus — golden 86s to 187s, full suite 107s to 306s). Outputs and failure sets are unchanged; interpreter performance (closure compilation, first-token prefilters) can be improved later without touching semantics.

Test plan

  • python -m pytest tests/ from interscript-py (18 pre-existing failures only; set diffed byte-identical against a pristine origin/main run)
  • python scripts/parse_parity_py.py --maps-root <maps-checkout> — 434/434 canonical, 434/434 raw (run pre-swap with --hand-rev HEAD; post-merge, pass --hand-rev <parent-of-swap>)
  • node scripts/parse-parity.mjs --py-root <this-branch> in an interscript/maps checkout — 411/434, 0 status flips vs the hand-py incumbent
  • ruff check src scripts tests (touched files)

Replace the hand-written recursive-descent ISC parser (the _Parser half
of interscript-py's isc.py) with a grammar-driven one, mirroring
interscript-ts's WO17 step-4b milestone (ts PR #128): the canonical
interscript-ruby .parg text (vendored verbatim in
grammar/grammar_source.py, sha256-pinned
b0f7a7caf462a1671a80ef974a59425c636f1d32cb3e405280991b21a6edd712 - the
same pin the ts vendored copy carries, synced by
scripts/sync_isc_grammar.py) is compiled by a PARG reader
(grammar/parg_reader.py), executed by a table-driven PEG interpreter
reproducing parsanol's capture/flattening semantics (grammar/peg.py),
and lowered to an IscDocument by an action layer (grammar/actions.py)
that drives the unchanged W1/W2 expansion (isc._Expander). parse_isc's
public API is unchanged; isc.py keeps the document model, the
expansion machinery and the compiled-map converter.

Evidence (all runs on this branch, maps at origin/main 05f5cc3,
428 maps + 6 libs = 434 files):

- documents: the new py-vs-py parity driver
  (scripts/parse_parity_py.py) parses every corpus file with BOTH
  parsers and diffs them: 434/434 identical under the maps canonical
  schema (scripts/parse-parity.mjs's py normalizer) AND 434/434
  field-for-field raw equality, including byte-exact from_source
  override keys and the written constraint order
- harness: maps scripts/parse-parity.mjs with this parser on the py
  side reports 411/434 vs ts - the same score, the same 23 failing
  files (0 status flips), and the same divergence-kind counts as the
  hand parser on the same-day ts side (the pre-existing gap is the py
  normalizer's constraint re-sorting and none-lowering, documented in
  the harness)
- suite: pytest 18 failed / 394 passed / 8 skipped; the failure set is
  IDENTICAL to a pristine pre-swap run (same 18 test ids: 4
  conformance + 14 embedded-map rows, all in parser-independent
  paths); the golden additivity gate still replays its rows
  byte-identically
- lint: ruff clean over the touched files

Four grammar-table extensions (GRAMMAR_EXTENSIONS, one named block),
each spelled in the grammar language and corpus-proven dormant:
any_arg item fallback, block-rule outside-brace when-guard, hyphenated
run dependency names - the same three the ts generated parser ships -
plus rule_items accepting `use` inside parallel/sequence blocks (the
py suite pins use-in-pass splicing; a corpus-wide scan finds none).
Hand-parser behaviors the grammar rejects are documented per site in
grammar/actions.py (invalid backslash-u escapes, hyphenated
identifiers outside run targets, multi-digit capture refs,
stage-level reverse/strip/swapcase, and the metadata raw-form quirks);
all corpus-dormant per the parity driver.

Known tradeoff, measured on this machine: the table-driven interpreter
parses the 434-file corpus in ~86s where the hand parser took ~7s
(typical maps still parse in tens of ms; the gap shows only on the
megabyte jamo/hanja maps and in the gates that parse the whole
corpus - golden 86s -> 187s, full suite 107s -> 306s). Failure sets
and outputs are unchanged; closing the gap (closure compilation,
first-token prefilters) can proceed later without touching semantics.
@ronaldtse
ronaldtse merged commit 80bee0a into main Nov 11, 2026
13 checks passed
@ronaldtse ronaldtse mentioned this pull request Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant